[ET Device Support] Add device tensor helper functions to TensorPtr API by Gasoonjia · Pull Request #18761 · pytorch/executorch

Gasoonjia · 2026-04-08T05:31:52Z

Stack from ghstack (oldest at bottom):

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
[ET Device Support] Extract shared device test utilities to reduce redundancy #18762
-> [ET Device Support] Add device tensor helper functions to TensorPtr API #18761
[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
[ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
[ET Device Support] Define AOT device copy ops registry #18728
[ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
[ET Device Support] Module: allocate device memory for planned buffers #18476
[ET Device Support] MemoryManager: add per-buffer device metadata #18475
[ET Device Support] MethodMeta: expose per-buffer device placement API #18474
[ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
[ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
[ET Device Support] Device-aware memory planning: separate buffers per device type #18375
[ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
[ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
[ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
[ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
[ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
[executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

Add clone_tensor_ptr_to_device and clone_tensor_ptr_to_cpu to tensor_ptr.h for cloning tensors between host and device memory via DeviceAllocatorRegistry.
Extend the existing make_tensor_ptr(const TensorPtr&, ...) overload with optional device_type/device_index parameters (default CPU/0) for seamless device placement.

Differential Revision: D99913077

Add clone_tensor_ptr_to_device and clone_tensor_ptr_to_cpu to tensor_ptr.h for cloning tensors between host and device memory via DeviceAllocatorRegistry. Extend the existing make_tensor_ptr(const TensorPtr&, ...) overload with optional device_type/device_index parameters (default CPU/0) for seamless device placement. Differential Revision: [D99913077](https://our.internmc.facebook.com/intern/diff/D99913077/) [ghstack-poisoned]

pytorch-bot · 2026-04-08T05:31:56Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18761

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 71 New Failures, 1 Cancelled Job

As of commit 888fdcf with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

Build Presets / apple (pybind) / build (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / windows (pybind) / build (gh)
Process completed with exit code 1.
Cadence Build & Test / cpu-test / test-aot / test-aot (gh)
backends/cadence/aot/tests/test_replace_ops_passes.py::TestReplaceOpsPasses::test_replace_transposed_conv_with_linear_0
periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 6a3876268df74955de25fe7b2b48259b296f23dd71ffbb0cce1dce77997463eb /exec failed with exit code 1
periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 3e66d87ca6982c1364240cffb99fce9900061f9eff75539a0eb841aa06eaae67 /exec failed with exit code 1
periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 439b161a27489b84f51116a51562d1d164caac6c33e06f8e95c2d338fd3d8efd /exec failed with exit code 1
pull / android / build-android (gh)
pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux-android / linux-job (gh)
RuntimeError: Command docker exec -t 37d06b7c399ccc34e3341e4bcaea838d919e6564ae00420ce00b05c63c015164 /exec failed with exit code 2
pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-lora-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-lora-multimethod-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-mcu-cortex-m-backend / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 7b51154770e63bc0d556b75f1bb4c65e56ded6681374ffb10fb05ef4ac9ccbbe /exec failed with exit code 1
pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 947db4285bba47f30eada217ee463a96cbf350d31307b824c99915970d66b59e /exec failed with exit code 1
pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t bc356214b5eea24940bbae5297e32c01f5cf3e84fc286e881b477bfb863a50e6 /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 75b1a6ce7cdb3b3c826e6107f20f772d24bdb219ecc2415a1bee18072f5e16d8 /exec failed with exit code 1
pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t faee72a0c09fb71ceee8a5f19c08a94b98174cbb2614c3044a6fc6169277d60d /exec failed with exit code 1
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t c51e26ef27652f6b6437a6d096a8372dba90a952ff1a2ccc6e31ee22abec1d23 /exec failed with exit code 1
pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 56c2b5269d0698bb0ba586dab55d5945e2b6252dcd7cff1397ee5b5436ca3e22 /exec failed with exit code 1
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 90f38fe8f9bf13b4d4c21b6bb4509af02f3be21a8e5406f842ac73b258b2c761 /exec failed with exit code 1
pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 9d43acb81d2c64df2745caccbf9a7436f9426611c24a22001f92149bfc04c9d6 /exec failed with exit code 1
pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 1d51b177351c9842ce0a8ee932ef80caef8a06881d26558dacc324d14e464d8f /exec failed with exit code 1
pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 017bdbaa372cbf0bc5bfd2dee7cb69983516f02098d55e72df0c2e8377f36091 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 231061e55cf37c73d91f0ea6a231c717602f13b2ab39bf1c1b781b0595461f23 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 624d67f8717bcc8e8b53c98ebe02d9e627f6473870930c3d0e196968810b2e16 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 354c725fe632d4e0504b29b9d6c6a935fb6aed95b9e1564bde80a3ee20ac64ca /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 2e14395a8cf27f5a7f6e6aa8fcbb6e5a7442a3aaf174a2bc184218b4651bcb8d /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 75d33883395b8a6629bd07228fc7f606ebb1415ea6cbea6c5f789c0bc2845a9d /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t 0fc162cea56ab6a60d5650d29238d1912f8e24a26376f6182a4fad0f3e8523b5 /exec failed with exit code 1
pull / test-moshi-linux / linux-job (gh)
test_mimi
pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
RuntimeError: Command docker exec -t 378aa885c7449e77cd33dc657180fac11ec8aaaa06c497bfd7e2fb6dcfdc008c /exec failed with exit code 1
pull / test-openvino-linux / linux-job (gh)
RuntimeError: Command docker exec -t 64e8a0ece75e0b023b5d4dfc3913c27e86b373e9d1388d71e0df603fe3da2864 /exec failed with exit code 1
pull / test-parakeet-xnnpack-linux / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-phi-3-mini-runner-linux / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
RuntimeError: Command docker exec -t 7b53ebf5cf5056e7727aebd55be688d45e935c7c6f87708a7c8f7e2ccf62e8c8 /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
RuntimeError: Command docker exec -t b04c09f4ae4c8ea34c612df865688f10a528f34ab7649fddc329c6c3e56a5e33 /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
RuntimeError: Command docker exec -t 5d99e1a2efc1d318e2e98174cd9188d07c5d21d5dd22a93263a97269d220a80d /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh)
RuntimeError: Command docker exec -t a1576e1604bea71460a84883e706ffaa1a8e7390c03e002503cb03a0e9c6a2fe /exec failed with exit code 1
pull / test-samsung-models-linux / linux-job (gh)
test_w2l_fp16
pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-vulkan-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t a8ab7cc0b7fa32b831700ab73833f9c9ec3173b00a555f1305294c64b963681a /exec failed with exit code 1
pull / test-vulkan-operators-linux / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / unittest / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t b1213f4fa171a6e18ad459f5cb108bd777413e85604af703bdf4c8c48f178b19 /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
RuntimeError: Command docker exec -t 4bdf094ca2f332ab395562e1f45f042c3bf9ebf35b8adfc93b1151b4d99856bc /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 21a016156fbdc55d88f9a248d2be59b51f9b652f360f2fb89122be382798eb06 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-nxp-neutron / linux-job (gh)
RuntimeError: Command docker exec -t 23c1705651316f04b48f0b1a16262f9bc8e6a10a1b91514ae65dea1560bb56df /exec failed with exit code 2
pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
RuntimeError: Command docker exec -t 164e391addabc04133afb88d9c5798029106a8ee9afeed42ef79cf81d787a986 /exec failed with exit code 2
pull / unittest-wasm-bindings / linux-job (gh)
RuntimeError: Command docker exec -t 6102dc56762eb953095dce0d3d86bde24fa2fd4b750e847335156ff3a20e6eb8 /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`

CANCELLED JOB - The following job was cancelled. Please retry:

Test CoreML Backend / test-coreml / test-backend-macos (coreml, operators) / macos-job (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-04-08T05:32:29Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…TensorPtr API" Add clone_tensor_ptr_to_device and clone_tensor_ptr_to_cpu to tensor_ptr.h for cloning tensors between host and device memory via DeviceAllocatorRegistry. Extend the existing make_tensor_ptr(const TensorPtr&, ...) overload with optional device_type/device_index parameters (default CPU/0) for seamless device placement. Differential Revision: [D99913077](https://our.internmc.facebook.com/intern/diff/D99913077/) [ghstack-poisoned]

digantdesai

Review automatically exported from Phabricator review in Meta.

Gasoonjia requested a review from shoumikhin as a code owner April 8, 2026 05:31

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 8, 2026

meta-codesync bot added fb-exported meta-exported labels Apr 8, 2026

Gasoonjia mentioned this pull request Apr 9, 2026

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788

Open

digantdesai approved these changes Apr 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ET Device Support] Add device tensor helper functions to TensorPtr API#18761

[ET Device Support] Add device tensor helper functions to TensorPtr API#18761
Gasoonjia wants to merge 2 commits intogh/gasoonjia/162/basefrom
gh/gasoonjia/162/head

Gasoonjia commented Apr 8, 2026 •

edited

Loading

Uh oh!

pytorch-bot bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 8, 2026

Uh oh!

digantdesai left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Gasoonjia commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18761

❌ 71 New Failures, 1 Cancelled Job

Uh oh!

github-actions bot commented Apr 8, 2026

This PR needs a release notes: label

Uh oh!

digantdesai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Gasoonjia commented Apr 8, 2026 •

edited

Loading

pytorch-bot bot commented Apr 8, 2026 •

edited

Loading

This PR needs a `release notes:` label